1. Field of the Invention
The present invention relates to a technique of displaying an image.
2. Description of the Related Art
Mixed reality, that is, so-called MR is recently known as a technology of merging a real world and a virtual world seamlessly in real time. In one MR technology, a video see-through HMD (Head Mounted Display) is used and an object almost matching an object observed from the pupil position of an HMD wearer is sensed by, for example, a video camera. A CG (Computer Graphics) image is superimposed on the sensed image to generate an image. The generated image is presented in front of the eyes of the HMD wearer.
FIG. 12 is a block diagram showing the functional arrangement of a general video see-through mixed reality system (to be referred to as a system hereinafter). A general system will be described below with reference to FIG. 12.
As shown in FIG. 12, the system comprises an image processing apparatus 1202 and a video see-through HMD 1201.
The video see-through HMD 1201 includes an image sensing unit 1203, a display unit 1204, a three-dimensional (3D) position sensor 1205, and a I/F (interface) 1206. The image processing apparatus 1202 is generally an apparatus such as a personal computer or workstation with an advanced calculation processing function and graphic display function, and includes an I/F 1207, position and orientation information generation unit 1208, and CG rendering composition unit 1210.
The components forming the video see-through HMD 1201 will be described first.
The image sensing unit 1203 senses an image (physical space image) of the outside world which can be seen, in almost the same direction as a direction of line of sight, from a position almost matching the viewpoint of the wearer who is wearing the video see-through HMD 1201 on the head. The image sensing unit 1203 is generally provided for each of the right and left eyes to generate a stereoscopic image, and includes an image sensing element, an optical system, and a DSP (Digital Signal Processor) that executes an image process.
The display unit 1204 displays an MR image output from the image processing apparatus 1202. The display unit 1204 is also provided for each of the right and left eyes, and includes a display device and optical system. A small liquid crystal display device or a retina scan type device by MEMS (Micro Electro Mechanical System) is used as the display device.
The 3D position sensor 1205 measures the position and orientation of itself. A magnetic sensor or a gyro sensor (acceleration and angular velocity) is used as the 3D position sensor 1205.
The I/F 1206 interconnects the video see-through HMD 1201 and the image processing apparatus 1202. The image sensed by the image sensing unit 1203 and a measurement result by the 3D position sensor 1205 are transferred to the image processing apparatus 1202 via the I/F 1206. The MR image generated on the image processing apparatus 1202 side is input to the video see-through HMD 1201 via the I/F 1206. The I/F 1206 that is required to transmit an enormous quantity of data in real time uses a connector compatible with a metal line such as a USB or IEEE1394 or an optical fiber such as GigabitEthernet®.
The components of the image processing apparatus 1202 will be described next.
The I/F 1207 interconnects the image processing apparatus 1202 and the video see-through HMD 1201. The image sensed by the image sensing unit 1203 and the measurement result by the 3D position sensor 1205 are transferred to the image processing apparatus 1202 via the I/F 1207. The MR image generated on the image processing apparatus 1202 side is input to the video see-through HMD 1201 via the I/F 1207.
The position and orientation information generation unit 1208 calculates position and orientation information representing the position and orientation of the viewpoint of the video see-through HMD 1201 wearer based on the measurement result by the 3D position sensor 1205 which has been received from the video see-through HMD 1201. Alternatively, a method of generating the position and orientation information of the viewpoint by using a marker or the like within the image sensed by the image sensing unit 1203 may also be used, as a matter of course.
Contents 1209 form a storage device which stores data pertaining to virtual objects forming the virtual space.
The CG rendering composition unit 1210 generates an image (CG image), of the virtual space according to the data of the contents 1209, which is seen from the viewpoint having the position and orientation represented by the position and orientation information generated by the position and orientation information generation unit 1208. The CG rendering composition unit 1210 composites the CG image on the image received from the video see-through HMD 1201 via the I/F 1207, and generates an MR image. The CG rendering composition unit 1210 outputs the generated MR image to the video see-through HMD 1201 via the I/F 1207.
With the above-described arrangement, the video see-through HMD 1201 wearer can experience a mixed reality world obtained by merging a real world and a virtual world seamlessly in real time.
In the above-mentioned system using the video see-through HMD, it is important to secure sight of the wearer, even if a communication error occurs between the HMD and the image processing apparatus. In particular, when a wireless scheme is employed for image transmission, errors may occur frequently depending on the environment of use and the distance between the apparatuses. In one technique, when such communication error occurs, a frame of a predetermined time before a carrier wave disappears in infrared ray video transmission is displayed as a still image together with a warning message (patent reference 1).
The following technique has also been proposed. That is, upon occurrence of a decoding error in one channel, a stereoscopic video display system including a plurality of cameras controls to transfer some or all of image data stored in another channel to the channel in which the error has occurred. The system interpolates some or all of image data in which the error has occurred, and displays the interpolated image (patent reference 2).    [Patent Reference 1] Japanese Patent Laid-Open No. 5-76078    [Patent Reference 2] Japanese Patent Laid-Open No. 7-322302
However, the above-described prior art has the following problems.
Assume that an error occurs in wireless image transmission and it is difficult to transmit the image. If an image of the successfully received, immediately preceding frame is displayed, a wearer does not know the actual outside world when moving. That is, it is difficult to secure the user's sight.
Furthermore, when an interpolation process is executed as in another conventional example, as the amount and frequency of error increase, it becomes difficult to reconstruct an image even if an identical image, stereoscopic image, or a preceding frame is used. Therefore, it is also difficult to secure the wearer's sight.